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"My belief is that we surely do have enough knowledge, evidence, and experience now to 
make things better than we are doing at the present time. . . . This is the time when we 
simply have to make the synthesis between research, existing best practice experience, to 
take the richness of recent innovations and make a serious national effort. " 

David Hamburg, M.D., president of the Carnegie Corporation 



''Children grow up in situations th^y didn't create but are reacting to. We try to intervene 
with them and to measure the chariges as if the children are the problem. But adults are 
responsible. " 

Paul Bracy, M.Ed., director, Offiice of Violence Prevention, 
Massachusetts Department of Publicliealth- 



"How do we, in the formative phases of an area such as violence prevention, foster 
programs that will be worthy of evaluation and develop methodologies that can eventually 
demonstrate outcomes, without killing the buds that might flower in the future?" 



Deborah Prothrow-Stith, M.D., assistant dean for government and community programs, 
Harvard School of Public Health 
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EXECUTIVE SUMMARY 



By any measure, violence is a significant public health problem. Adolescents, particularly 
African American males and those living in poverty, are at the greatest risk for b^ng either 
the victim or the perpetrator of an act of violence. This growing problem is threatening the 
safety, the health, and the lives of America's next generation of adults. 

In February 1990, Carnegie Corporation of New York funded Education Development 
Center, Inc. (EDC), to investigate the state of the art of violence prevention programs for 
young adolescents (ages 10-15 years). By examining the recent proliferation of intervention 
programs aimed at preventing or reducing violent behavior among this population and 
identifying those that have been evaluated, our purpose was to generate thoughtful 
recommendations for next steps. 

The project was informed by and designed to build upon past Corporation activities in this 
area. The Corporation's "Conference on Adolescent Violence: Research and Public Policy," 
held in February 1987, brought together researchers, service providers, and public officials 
to examine the problem of adolescent violence from these three perspectives and to 
collaborate in setting priorities for programs of research and service. A major 
recommendation of that group was that it would be scientifically more valid and socially 
more useful to design programs for a broader population than the future violent adolescent: 
"Programs designed for an entire grade or school or neighborhood would seem likely to be 
more effective than those seeking to pick out potentially violent individuals from among their 
peers. " Encouraging prosocial behavior in addition to eliminating antisocial behavior was 
recommended. 

"Prevention of Violence in Young Adolescents," the Carnegie Council on Adolescent 
Development's May 1989 workshop, reviewed the scientific evidence on the causes of 
violence-biological, psychological, and social--and began to explore a range of possible 
interventions. Many programs were discussed during the two-day meeting. However, there 
was insufficient time for an in-depth review of evaluation findings and the workshop was not 
devoted specifically to the development of programmatic goals and methods for intervention. 

To continue to build on the substantial foundation laid by these two previous meetings, it was 
necessary to examine in more depth the evaluation of violence prevention programs. 
"Violence Prevention for Young Adolescents: A Review of the State of the Art," the 
working conference held on July 12-13, 1990, allowed individuals "on the front lines" of 
violence prevention theory, practice, and policymaking to share their expertise and 
experiences with evaluation, and to discuss and debate existing knowledge, current practice, 
and other important issues. The recommendations informed by the conference, and by the 
survey of violence prevention programs carried out in preparation for the meeting, address 
programmatic issues, such as the identification of existing programs for which expanded 
evaluations should be funded and that can serve as laboratories for the development of new 
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evaluation techniques. In addition, specific products were recommended, such as a handbook 
on violence prevention evaluation to be developed collaboratively by practitioners and 
evaluation researchers and activities, and the convening of an annual meeting of violence 
prevention program practitioners. 



CONCLUSIONS AND RECOMMENDATIONS 

Between 1987 and 1990, staff at EDC participated in two reviews of the state of the art in 
violence prevention interventions: Injury Prevention: Meeting the Challenge and the 
background papers prepared for the CDC's Youth Violence in Minority Communities: A 
Forum on Setting the Agenda for Prevention (see page 9). The Carnegie conference on 
violence prevention for young adolescents, however, was perhaps the first attempt to 
systematically address the question of how these programs have been evaluated. As such, 
the project, and the two resulting Council working papers,^ deal primarily with the process 
and content of the evaluations themselves. Much of the conference discussion focused on 
what could be learned from these evaluations to inform and improve future efforts. 

During the conference another and different question was posed by Dr. Elena Nightingale, 
special advisor to the president of Carnegie Corporation. Of all of the programs 
investigated, she asked, were there any that in whole or in part warrant greate^' 
dissemination? Which of these programs, she was asking in effect, had been subjected to 
rigorous process and outcome evaluation and been found to be effective in preventing 
violence among young adolescents? 

The direct answer is that none of the programs surveyed has been so evaluated. Therefore, 
given the limitations of existing evaluations of violence prevention programs, it would be 
premature to come to closure about what works and what doesn't. That does not mean that 
we lack any indicators of which programs are promising. Nor does it mean that we should 
abandon programs that are currently reaching large numbers of persons. It does raise the 
quer,tion of what steps can be taken to improve our understanding of these programs and 
their effectiveness. And it poses two central challenges for violence prevention practitioners, 
researchers, and funding agencies: how to identify current, promising programs and 
strengthen their efforts while simultaneously designing new programs so well conceived and 
carefully implemented as to be worth the expenditure of resources for the kind of full-scale, 
rigorous evaluation that has been so rare. 



' This paper is complemented by a separate Council Working Paper focused on the survey 
of violence prevention programs carried out in preparation for the Carnegie Conference on 
Violence Prevention for Early Adolescents. See Wilson-Brewer R, Cohen S, O'Donnell L, and 
Goodman IF. (1991). Violence Prevention for Early Teen^: TheStateof the Art and Guidelines 
for Future Program Evaluation. Washington, D.C.: Carnegie Council on Adolescent 
Development. 
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To search for answers to these questions within the activities described in this paper alone 
would be insufficient. Evaluation is an important component-indeed, a critical one-for any 
project focused on achieving social change. But the questions we must answer are about 
design and implementation of approaches as well as evaluation. 

To truly understand both whether a violence prevention program was effective and the 
likelihood that it could be replicated, we must know the following: 

• How was the program developed? What theoretical or empirical body of 
knowledge informed its creation? 

• How were the intervention(s) and target populations selected? What 
indications existed that the fit between intervention and population were the 
most appropriate? Wliat, for example, was the role of formative evaluation? 

• What were the program's objectives and how were they carried out? What 
was the role of process evaluation? 

• . What effects did the program produce? What was the role of outcome 

evaluation? 

Taken together, the Carnegie conference and the Centers for Disease Control's forum (see p. 
9) provide the broader view that is necessary. In addressing Dr. Nightingale's questions and 
in providing the most useful context for the Carnegie conference's specific recommendations 
reported below, we will draw upon some of the points raised in the materials prepared for 
the CDC forum. 

In addressing the application of principles of community-based programs to violence 
prevention in minority communities, the EDC background paper drew upon several sources. 
These included the general injury prevention program model described in Injury Prevention: 
Meeting the Challenge, a model for diffusing health information in minority communities,^ 
and CDC's Planned Approach to Community Health Program (PATCH).^ 



^ National Heart, Lung, and Blood Institute. (1987). Strategies for Diffusing Health 
Information to Minority Populations: A Profile of a Community-Based Diffusion Model. 
Washington, D.C.: U.S. Department of Health and Human Services. 

^ Centers for Disease Control, (n.d.). Planned Approach to Community Health. Atlanta: 
U.S. Public Health Service. See also: Nelson CF, Kreuter MW, Watkins NB, and Stoddard 
RR. (1986). A Partnership Between the Community, State, and Federal Government: Rhetoric 
or Reality? IrttemationalJoumal of Health Education', 5:27-31. 
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The process for developing community violence prevention programs described as 
background for the CDC forum includes a series of elements: developing leadership, 
identifying the problem, devising a strategic plan to address the problem, translating the plan 
into action by designing an intervention or program of interventions, implementing that 
intervention successfully, and evaluating its effects/ 

It should be stressed-and the point was emphasized by participants in the Carnegie 
conference-that although evaluation is always named last in the series of steps (e.g., design, 
implementation, and evaluation), it must be seen as an integral part of the process. "The 
time to plan an evaluation," concluded the National Committee for Injury Prevention and 
Control, "is in the very earliest stages of program design."^ 

We must, however, return to the initial question: What can the evaluations of existing 
violence prevention programs tell us about what directions are promising? If the evaluations 
alone are insufficient to identify promising programs, then what else do we need to know to 
improve our understanding and move the field forward? 

In its 1987 survey of the state of the art in injury prevention, the National Committee for 
Injury Prevention and Control studied a wide range of existing countermeasures against all 
forms of injury, most of which had not been rigorously evaluated. In categorizing these 
interventions, the committee used the following terms: "proven effective," "promising," 
"ineffective," or "ufiknown or insufficiently studied." For each intervention, a 
recommendation included both a statement of efficacy and suggestions for further use. Thus, 

• an intervention that is proven effective (e.g., bicycle helmets) should be used 
and the program should be routinely monitored (i.e., a process evaluation is 
necessary, but not an outcome evaluation) 

• an intervention that is promising (e.g., raising excise taxes on alcohol to 
reduce its availability) should be used, monitored (i.e., through process 
evaluation), and its outcomes should be rigorously evaluated 

• an intervention that is ineffective or counterproductive (e.g., painted lines at 
crosswalks) should not be used 



Cohen S and Lang C. (1990). Application of the Principles of Community-based 
Programs. A background paper prepared for the Forum on Youth Violence in Minority 
Communities; Setting the Agenda for Prevention, p. 6. Available from the Division of Injury 
Control, CDC. 

^ National Committee for Injury Prevention and Control. (1989). Injury Prevention: 
Meeting the Challenge. New York: Oxford University Press, p, 78. 
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• an intervention whose efficacy is unknown or insufficiently studied (e.g., 
designated driver and safe-ride programs) should be the subject of further 
research^ 

The categories used by the committee were deliberately broad because the evidena*. available 
on which to make judgments about interventions was often extremely limited. In many 
cases, only a single article (or unpublished document) existed about a specific intervention. 
Committee working groups worked hard to find a reasonable basis on which to assign each 
intervention to a category. The number of proven effective or completely ineffective 
interventions was small. Much remains to be learned about the others. 

An attempt to describe the 51 projects surv^eyed in these terms reveals that the vast majority 
are of unknown or insufficiently studied value in preventing violence among young 
adolescents. As most of these projects have not been evaluated, this conclusion is not 
surprising. 

However, based upon existing— albeit incomplete— evaluation data, it is possible to identify a 
smaller number of projects whose assessments suggest that 1) their results to date have been 
promising, and (2) further study could yield valuable information about the development, 
implementation, and evaluation of future programs. These projects include 

• The Paramount Plan: Alternatives to Gang Violence (Paramount, California) 

• Project Stress Control School-Based Curriculum (Atlanta) 

• Resolving Conflict Creatively Program (New York City) 

• Second Step: A Violence Prevention Curriculum (Seattle) 

• Viewpoints Training Program (Santa Barbara, California) 

• Violence Prevention Curriculum Project (Newton, Massachusetts) 

• Violence Prevention Project (Boston)^ 

These projects should be continued and should be monitored through process evaluation and, 
most important, new or expand )d outcome evaluations. At the same time, a study of their 
conceptual, theoretical, and research bases should be undertaken to answer a series of 



^ National Committee for Injury Prevention and Control, op. cit., p. 113. 

^ Descriptions of these projects are contained in Wilson-Brewer R, Cohen S, O'Donnell L, 
and Goodman IF. op. cit. 



questions: What are the behaviors that the programs are intended to change? What are the 
probable or known causes of those behaviors and to what extent did that research base 
influence the selection of interventions and target populations? Which specific considerations 
informed the selection of interventions? To what extent does the community in which the 
program operates feel any sense of ownership of violence prevention, in general, and the 
program specifically? 

Only by answering such questions will it be possible to ideiitify model projects that could be 
disseminated widely without the need to conduct a full outcome evaluation in each new 
setting. And, only by asking such questions will it be possible to close the gap-identified by 
several-between violence prevention research in the behavioral sciences and practice in the 
field. As commented Ron Slaby, we wouldn't think of dealing with an AIDS prevention 
program, at this point without being informed of the research base about how AIDS in 
contracted. With AIDS, our practice is guided by the state of research evidence. That is 
often not the case with regard to violence prevention. 

With these considerations in mind, we present the following recommendations for action. 
They are of two types. The program recommendations were developed by the authors during 
a reanalysis of the survey data. The products and activities, on the other hand, were 
suggested by the working groups and refined during subsequent plenary discussion. 

PROGRAM RECOMMENDATIONS 

1. Funding and technical assistance should be made available to review the programs 
listed above with a view to determining which among them are sufficiently promising 
to make the expenditure of resources for new or additional outcome evaluations 
worthwhile. 

2. The provision of resources for the expanded evaluation of promising programs is a 
task beyond the capacity of any single funding source. Because this need is both 
national and resource-intensive, the effort should be coordinated by a consortium of 
government agencies and foundations. 

3. The identified promising projects should be used as laboratories for the development 
of intermediate outcome measures, for the validation of existing measures, and for the 
development of new evaluation methods. Such efforts will require the collaboration 
of program practitioners, individual researchers, and public and private research 
institutions. 



SPECIFIC PRODUCT RECOMMENDATIONS 

1 . A handbook on violence prevention evaluation should be developed as a collaborative 
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effort of evaluators and practitioners. Such a volume is necessary to aid violence 
prevention practitioners and to advance the state of the art. Such a handbook would 
assist program staff to carry out evaluation tasks, as well as to collaborate with 
evaluation research specialists in the development and implementation of evaluation 
designs. It should contain information on available resources, including a list of 
evaluation researchers and their specific areas of expertise. General purpose primers 
on evaluation fail to provide good outcome and process measures that are relevant to 
violence prevention programs or identify qualitative evaluation strategies. A useful 
model for such a handbook is Evaluating AIDS Prevention Programs, published by 
the National Research Council.^ 

2. A primer and catalog on the identification and use of culturally sensitive measures in 
conducting formative and outcome evaluations should also be created. This document 
will review the evidence on how cultural differences in values and behavioral norms 
among ethnic and racial groups can affect the design and implementation of violence 
prevention interventions. It will identify measures that have been used successfully 
with different groups, and detail the critical steps in developing new measures. 

ACTIVITIES 

1. Convene an annual meeting of violence prevention practitioners to improve 
communication and problem solving. 

2. • Conduct a summer institute in program development and evaluation for violence 

prevention practitioners, to help them develop and enhance skills in carrying out 
evaluation tasks, and work effectively with specialists to plan evaluation designs. 

3. Develop interdisciplinary research centers to focus on the evaluation of violence 
prevention programs. 

4. Increase the outreach to and recruitment of minority students and faculty and provide 
scholarships to encourage study in the field of violence, especially in research and 
evaluation. 

5. Conduct rigorous evaluations of model programs already underway. 

Each of these recommendations speaks to a specific need that must be met if we are to 
improve the evaluation of violence prevention programs and, through improved evaluations, 
affect the development and implementation of more effective strategies. No single 
recommendation is sufficient, however. 



* Coyle SL, Boruch RF, and Turner CF (eds). (1991). Evaluating AIDS Prevention 
Programs (Expanded Edition). Washington, D.C.: National Academy Press. 
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If, as Dr. Hamburg suggested, there is a need to bring coherence to the nascent national 
effort to prevent violence among young adolescents, that is no less true of program 
evaluation. Increasing the capacity of program practitioners to learn from each other, to 
understand and benefit from evaluation research, and to collaborate fruitfully with 
researchers are related priorities. So too is the need for researchers to learn from violence 
prevention practice, not only to advance the development of theory, but to help develop new 
and innovative strategies for program evaluation. The identification of intermediary 
objectives-short-term gains that presage the longer-term goal of preventing violence-is a 
particularly critical need. How to organize these pressing tasks remains to be considered. 
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INTRODUCTION 



In late 1989, the Boston Globe reported that "When you go into a Boston high school and 
[ask] how many kids know of someone who has died of homicide, nearly all the hands go up 
in the room/^ Boston is not unique; the question would be answered similarly in many of 
America's urban areas. 

Epidemiologic evidence supports the anecdotal. According to the National Center for Health 
Statistics, over 30 years, death rates decreased in every age group except among 15-24 yeai* 
olds. For them, increased mortality was caused primarily by violence. Increasingly, the 
analogy is being drawn between life in America's poor urban neighborhoods and the battle 
zones of such counides as Lebanon, Northern Ireland, Afghanistan, and Burma.^^ The 
symptoms of post-traumatic stress syndrome are being observed among children for whom 
violence is the inescapable context of daily life. 

By any measure, violence is a significant public health problem. And adolescents, 
particularly African American males and those living in poverty, are at the greatest risk for 
being either the victim or the perpetrator. This growing problem is threate^ ' the safety, 
health, and lives of America's next generation of adults. 

There have been numerous attempts to address the threats of violence. For example. Healthy 
People 2000, which outlines health promotion and disease prevention objectives for the 
nation, calls for reductions in (1) homicide rates among African American men ages 15-34 
(from 90.5/100,000 to 72.4/100,000), (2) assaultive injuries among people age 12 and older 
(from 11.1/100,000 to no more than 10/100,000), and (3) the incidence of physical fighting 
and weapon carrying among adolescents ages 14-17 (both by 20 percent). However, doing 
so, the report acknowledges, will require extending "coordinated, comprehensive violence 
prevention programs to at least 80 percent of local jurisdictions with populations over 
100,000."^^ 

That America's history has been a violent one has long been accepted. However, 
acknowledgment that violence is a public health problem because of the enormous toll it 



Boston Globe. (October 31, 1989). p. 22. 

Stanley A. (June 18, 1990). Child Warriors. Time, p. 30-52. 

U.S. Public Health Service. (1990). Healthy People 2000: National Health Promotion 
and Disease Prevemion Objectives. Washington, D.C.: U.S. Department of Health and Human 
Services, p. 240. 
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takes in terms of morbidity and mortality is more recent. And acknowledgement that 
violence among youth is one of the nation's most pressing problems, one that requires 
innovative, well-designed intervention strategies, is even more recent. 

Adolescence can be an extremely difficult period-a fact to which many adults can attest. 
Add to the physical and emotional changes young people are experiencing the new demands 
that society is placing on them and it is not surprising that it is considered a time of turmoil. 
However, little attention has been paid to early adolescence (ages 10-15 years), a time of 
transition when many of the important developmental tasks associated with adolescence are 
often being accomplished or, at the very least, attempted. During this time, young people 
are seeking autonomy, independence, and separation from parents as new identities are 
formed.*^ "For many youth, early adolescence is one of the last real opportunities to affect 
their educational and personal trajectory."^'* V'hen one adds to this picture of early 
adolescence statistics on violence and its effects, what we see is an age group whose limitless 
potential is increasingly in jeopardy. 

Surely, the statistics are by now familiar. Still, they must be restated, however briefly. 
Homicide is the most devastating and permanent violence outcome. It is the 12th leading 
cause of death in the United States, at a rate of 10.6 per 100,000. However, it is the second 
leading cause of death for 15- to 24-year-olds; for African American males in that same age 
group, it is the leading cause of death. Homicide is not distributed evenly throughout the 
population. It takes its greatest toll among African Americans, males, and the young. 
Among children, those under three and over 14 years old are most vulnerable to murder. 



See Rosenberg M. (December 1988). Violence is a public health problem. Transactions 
and Studies of the College of Physicians of Philadelphia, 10(1-4): 147-68. Also Prothrow-Stith 
D, Spivak H, and Hausman A. (April 1987). Adolescent Violence: The Evaluation of Health 
Education as a Public Health Strategy. New England Injury Prevention Research Center 
Working Paper No. 4. 

McCandless BR. (1970). Adolescents: Behavior and Development. Hinsdale, IL: 
Dryden. See also: Csikszentmihalyi M, and Larson R. (1984). Being Adolescent: Conflict 
and Growth in the Teenage Years. New York: Basic Books. 

Jackson AW, and Hombeck DW. (1989). Educating Young Adolescents: Why We Must 
Restructure Middle Grade Schools. American Psychologist; 44(5):831-36. 

^^Rosenberg, ML, et al. (1987). Violence: Homicide, Assault, and Suicide. In Amler, RW 
and Dull, HB (Eds.), Closing the Gap: Tfie Burden of Unnecessary Illness. New York: Oxford 
University Press, 164-178. 

U.S. Department of Justice. (1980). Crime in the United States: FBI Uniform Crime 
Reports. Washington, D.C.: U.S. Department of Justice. 
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Other figures developed by the Bureau of Justice Statistics are starkly instructive. When the 
FBI estimated, in 1985, the lifetime risk of becoming a victim of homicide, the following 
ranking emerged (from least risk to greatest) 



These numbers reflect homicide statistics of more than a decade ago (1978-1980). While the 
risk has decreased somewhat overall in recent years, it has increased for African American 
males. UnpubUshed FBI data indicate that as of 1989 the lifetime risk of becoming a victim 
of homicide was 



More male adolescents die from gunshots than from all natural causes. In 1988, nearly half 
of all 15-19~year old African American males who died were killed by guns. Between 1984 
and 1988, gun-related deaths within this group of African American adolescents increased by 
100 percent.^* 

Much less information is available about nonfatal intentional injuries. Even the magnitude of 
the problem is unclear. For example, the FBI provides a figure for the total number of 
"serious assaults" in its Uniform Crime Reports. However, when researchers compared FBI 
assault data with hospital records for the same period, it was found that nearly four times as 
many assault injuries are recorded in hospital records than had been reported to the police. 



Langan PA, and Innes CA. (1985). The Risk of Violent Crime. Bureau of Justice 
Statistics Special Report {NCh97 119). Washington, D.C.: U.S. Department of Justice. 

Fingerhut LA, Kleinman JC, Godfrey E, and Rosenberg H. (1991). Firearm Mortality 
Among Children, Youth, and Young Adults 1-34 Years of Age, Trends and Current Status: 
United States, 1979-88. Monthly Vital Statistics Report; 39(11), supp\QmtnVA-l6. Hyattsville, 
MD: National Center for Health Statistics. 

Barancik JI, Chaterjee BF, Greene YC, Michenzi EM, and Fife D. (1983). Northeastern 
Ohio Trauma Study: I. Magnitude of the Problem. American Journal of Public Health; 
73(7):746-51. 



White females 
White males 
Nonwhite females 
Nonwhite males 



1 chance in 450 
1 chance in 164 
1 chance in 117 
1 chance in 28 



White females 
White males 

African American females 
African American males 



1 chance in 496 
1 chance in 205 
1 chance in 117 
1 chance in 27 
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Nor are physical injuries the only type of nonfatal outcome. Psychological damage is 
inadequately assessed and, subsequently, undervalued when estimating violence outcomes. 

In 1984, youth below the age of 16 were responsible for 59 percent of all cases handled by 
the juvenile courts. More specifically, this under-16 age group was responsible for 59 
percent of all crimes against persons, 63 percent of all property crimes, 41 percent of all 
public order crimes, and 62 percent of all status offense cases. Males were charged with 81 
percent of all delinquency cases; they were responsible for the majority of cases in all but 
one offense category: runaway.^" 

And in 1987, statistics indicate that more than 1.8 million 12- to 19-year-olds--again, the 
majority of them males-were involved in violent crimes; approximately one-third of them 
involved the use of a weapon.^* In a violent crime, a teen is more likely to be physically 
attacked than an adult-and far less likely to report the crime. Only one in three victims ages 
12-15 years old told police about the crime, compai-ed to half of all adults. And these young 
people often know their attacker. Approximately 45 percent recognized or knew the 
offender, with 83 percent of African American teen victims reporting African American 
assailants and 76 percent of white teen victims reporting white assailants.^^ 

At school, many children's greatest fear is aggression by other children. Those fears ?xc 
often well-founded. One-third of assaults against younger adolescents occur in school. In 
one study, almost half of the boys and one-fourth of the giris surveyed reported having been 
in at least one violent altercation during the past year, 34 percent reported having been 
threatened, 41 percent of boys and 24 percent of girls said they could easily obtxijn a 
handgun, 23 percent of boys reported carrying a knife, and three percent reported carrying a 
gun to school. 

The last national study of school violence^ conducted 12 years ago, found that 50 percent of 
assaults and 68 percent of robberies to 12- to 15-year-olds occurred in school. Typically, the 



2° Snyder HN, Finnegan TA, Nimick EH, Sickmund MH, Sullivan DP, and Tiemey NJ. 
(August 1987). Juvenile Court Statistics 1984. Washington, D.C.: U.S. Department of Justice, 
Office of Juvenile Justice and Delinquency Prevention. 

^' U.S. Department of Justice, Bureau of Justice Statistics. (March 1988). Report to the 
Nation on Crime and Justice. 2nd ed. Washington, D.C.: Government Printing Office. 

^2 Calhoun J. (1988). Violence. Youth and a Way Out. Washington, D.C.: National 
Crime Prevention Council. 

2^ American School Health Association. (1988). National Adolescent Student Health 
Survey. Washington, D.C.: American School Health Association, Association for the 
Advancement of Health Education, Society for Public Health Education. 



violence takes place as the result of an argument and students are known to cne another.^"* 
A more recent survey of teachers found middle schools to be the locus of significant levels of 
violence; teachers reported more physical violence among middle-schoolers than at the high 
school level.^ 

Still, reported instances of interpersonal violence in the schools, like arrest statistics, reflect 
only a small part of the problem, and most statistics do not distinguish between student and 
teacher victims. The U.S. Census Bureau's 1987 National Crime Survey round close to 
184,000 staff, students, and visitors were injured as a result of school crime.^^ 

We know the risk factors for violence involvement: being young, being male in a society 
that glorifies winning at any cost, being unemployed, being poor, being an African 
American, being the resident of an urban area, and having previously been a victim of 
violence. We also know v^hat many of the contributing factors are: lack of social support 
networks; poor aggression and conflict management skills; widespread availability of 
weapons, alcohol, and other drugs; and exposure to potentially life-threatening people and 
situations. 



THE ROLE OF CARNEGIE CORPORATION 

Recent studies indicate that certain violent behavior patterns appear early in childhood and 
that early intervention may prevent violent behavior in later years. As a result. Healthy 



National Institute of Education. (1978). Violent Schools-Safe Schools: The Safe School 
Study Report to the Congress. Washington, D.C.: U.S. Department of Health, Education, and 
Welfare. 

Office of Educational Research and Improvement. (1987). Public School Teacher 
Perspectives on School Discipline. Washington, D.C.: U.S. Department of Education. 

Foley D. (May 1990). Danger: school zone. Teacher Magazine, p. 5S. 

^'^ Although African Americans have disproportionately high rates of homicide, several 
studies suggest that homicide rates are strongly correlated with poverty and low socioeconomic 
status. When the homicide rates of African-American and white groups with the same 
socioeconomic status have been compared, researchers have found rates for both groups have 
been almost equal. See Centerwall BS. (1984). Race, socioeconomic status, and domestic 
violence, Atlanta 1971-72. American Journal of Public Health; 74:813-15. Lx)ftin C and HiU 
RH. (1974). Regional subcultures and homicide: An examination of the Castil-Hackney thesis. 
American Social Review; 39:714-24. Willin KR. (1984). Economic sources of homicide: 
Reestimating the effects of poverty and equality. American Social Review; 49:283-89. Parker 
RN and Smith MD. (1979). Deterrence, poverty, and type of homicide. American Journal of 
Sociology; 85:614-24. 



People 2000 calls for the teaching of nonviolent conflict resolution skills (preferably as part 
of a comprehensive school health education program) in at least 50 percent of the nation's 
elementary and secondary schools by the decade's end.^^ 

We hope to reduce violent behavior and increase violence prevention skills. At the same 
time, however, we must recognize the conclusion of Injury Prevention: Meeting the 
Challenge, the National Committee for Injury Prevention and Control's two-year study of the 
state of the art in injury prevention. "There are few models and much uncertainty about the 
effectiveness of many available interventions [against interpersonal violence]. Therefore, the 
greatest need is for interventions that are designed with specific, measurable objectives. 
Evaluations of these interventions should be widely disseminated."^^ 

It is time, therefore, to examine carefully the recent proliferation of intervention programs 
and materials aimed at preventing or reducing violent behavior. Progress toward 
"coordinated, comprehensive violence prevention" must be based on a review of existing 
evaluated programs and the thoughtful development of recommendations for next steps. That 
was Carnegie Corporation's purpose in convening the working conference "Violence 
Prevention for Young adolescents: A Review of the State of the Art." 

In February 1990, Carnegie Corporation of New York funded Education Development 
Center, Inc. (EDC), with headquarters in Newton, Massachusetts, to conduct the following 
activities over an eight- month period: 

1. Identify violence prevention programs for young adolescents (ages 10-15 
years) in the United States. 

2. Collect data about each program, including goals, target populations, major 
activities, and evaluation methodology and outcomes. 

3. Produce a workshop document that describes these programs, summarizes 
evaluation findings, critiques methodologies, and addresses such issues as 
barriers to effective program design, implementation, and evaluation. 

4. Convene an interdisciplinary group of violence prevention practitioners, 
violence and aggression researchers, program evaluators, and government 
representatives to discuss lessons learned and to collaborate in setting priorities 
for programs of service and research. 



U.S. Public Health Service. Ibid. 

2^ National Committee for Injury Prevention and Control. (1989). Injury Prevention: 
Meeting the Challenge. New York: Oxford University Press. 
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5. Prepare a final report for public distribution. 

Conference participants were selected from a wide variety of fields and areas of expertise 
(see Appendix A). Only by bringing together such a diverse group for discussion and debate 
can this critically important field advance. Clearly, there were many other individuals whose 
participation would have added much to the discussions. However, it was necessary to limit 
the total number of participants while also drawing on people from a range of regions and 
disciplines-and with a range of experiences and perspectives to share. 

This conference was informed by and designed to build upon a recent series of Carnegie 
Corporation activities. The first is the report of the Carnegie Council on Adolescent 
Development's Task Forc^ on Education of Young Adolescents. Entitled Turning Points: 
Preparing American Youth for the 21st Century, it examines all aspects of middle grade 
education and proposes a comprehensive set of reforms covering school organization, 
curriculum, classroom practices, health and counseling services, relations with families, and 
community linkages.^^ Although the report did not focus on violence prevention 
interventions, per se, it did acknowledge that violence has become a serious problem that 
schools cannot solve in isolation, and cited several programs that are employing new 
approaches to address the issue. The need for greater efforts and more resources was also 
noted, as was the association of crime and violence with poverty and limited opportunities. 

The Council reports Life Skills Training: Preventive Interventions for Young Adolescents^^ 
and School and Community Support Programs that Enhance Adolescent Health and 
Education,^^ in combination with an earlier related paper, Teaching Decision Making to 
Adolescents: A Critical RevieWy^^ also provided important insights for this current report. 
All three focused to some extent on the importance of violence prevention, although those 
words were not always used. 

For example, life skills training, as defined in the report cited above, is "the formal teaching 



Carnegie Council on Adolescent Development. (1989). Turning Poims: Preparing 
American Youth for the 21st Century. Washington, D.C.: Carnegie Council on Adolescent 
Development. 

Hamburg B. (1989). Life skills Training: Preventive Interventions for Young 
Adolesct,its. Washington, D.C.: Carnegie Council for Adolescent Development. 

Price R, Cioci M, Penner W, and Trautlein B. (1990). School and Community Support 
Programs that Enhance Adolescent Health and Education. Washington, D.C.: Carnegie 
Council on Adolescent Development. 

Beyth-Marom R, Fischoff B, Jacobs M, and Furby L. (1989). Teaching Decision Making 
to Adolescents: A Critical Review. Washington, D.C.: Carnegie Council on Adolescent 
Development. 



of requisite skills for surviving, living with others, and succeeding in a complex society."^ 
Social competence, a major category of life skills, involves nonviolent conflict resolution, 
friendship formation, peer resistance, assertiveness, and renegotiation of relationships with 
adults. Social support programs, operating in schools and community-based organizations, 
are designed to reduce the risk of both educational failure and poor health. Few of the 
programs reviewed in the report stated violence prevention as an explicit goal; however 
programs did address such violence-related goals as reducing discipline problems, reducing 
child abuse and neglect, and decreasing dropout rates. And effective decision-making skills 
(also a major category of life skills) are also essential for adolescents as they are presented 
with seemingly endless opportunities to take risks-many of which could affect their lives and 
the lives of those with whom they are in conflict-be they friends, family members, 
acquaintances, or strangers. 

Finally, several Council efforts, culminating in the May 1989 workshop, "Prevention of 
Violence in Young Adolescents," reviewed the scientific evidence on the biological, 
psychological, and environmental causes of violence, and began to explore a range of 
possible interventions. Many programs were introduced or discussed during the two-day 
meeting. However, there was not sufficient time for an in-depth review of evaluation 
findings and the workshop was not devoted specifically to the development of programmatic 
goals and methods for intervention. 

In these recent initiatives, the Carnegie Council on Adolescent Development has explored the 
theoretical bases of violence, a major category of interventions (life skills programs), and the 
middle school context in which much youth violence takes place. To continue along this 
path, it is necessary to integrate the results of these activities with the theoretical and 
practical expertise of individuals "on the front lines" of violence prevention theory, practice, 
ar*d policy making. 

In a time of many vexing and tragic societal problems, the need to prevent violence by and 
against young adolescents takes on a special urgency. The stakes are high. In truth, the 
future of a generation of young people, many of them African American males, is at risk, as 
are the communities in which they live and the nation as a whole. 

In a crisis, the pressure to act-to develop a new program or replicate an existing 
one— may be overpowering. In such a climate, evaluation may be an afterthought or, more 
frequently, not a thought at all. Yet, it is under precisely such circumstances, that is, when 
the need to respond is great, that program evaluation is most important. Otherwise, 
ineffective programs will be replicated widely based on an appearance of success, thereby 
reducing the impetus and the resources that should be devoted to designing and evaluating 
better interventions. 



^ Hamburg B. op. cit., p. 3. 
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It was the purpose of this Camegie-sponsored conference, therefore, to take stock of what 
has been learned thus far from existing violence prevention programs and to consider 
especially the barriers to conducting good process and outcome evaluations and how those 
barriers might be overcome. 



THE FORUM ON YOUTH VIOLENCE IN MINORITY COMMUNITIES: 
SETTING THE AGENDA FOR PREVENTION 

This working paper also takes note of a separately organized but related "Forum on Youth 
Violence in Minority Communities: Setting the Agenda for Prevention." The forum was co- 
sponsored in December 1990 by the Centers for Disease Control and the Minority Health 
Professions Foundation with the Morehouse School of Medicine. Held in Atlanta, Georgia, 
December 10-12, 1990, the forum brought together 110 participants from public health, 
criminal justice, social service, academic organizations, foundations, and minority 
communities. There is, however, a direct link between the Carnegie Corporation and CDC 
gatherings. In preparation for the CDC conference, and with partial support from the 
Camegie Corporation, EDC staff prepared six background papers to guide the discussion, 
one of which, on the evaluation of violence prevention programs, drew extensively on the 
results of the Camegie Conference.^^ 

The forum has been described in detail in Public Health Reports,^^ and the background 
papers prepared by EDC are available from the Centers for Disease Control. Consequently, 
we summarize the forum briefly here. 

The forum proceeded from the following critical assumption: 

Despite an urgent need, there is little guidance currently available for 
communities that wish to develop their own violence prevention programs. 
The absence of useful guidance is attributable to three factors. First, we know 



The CDC conference papers, which are available from the Division of Injury Control, are: 
Wilson-Brewer R and Jacklin B. Violence Prevention Strategies Targeted at the General 
Population of Minority Youth; Northrop D, Jacklin B, Cohen S, and Wilson-Brewer R. Violence 
Prevention Strategies Targeted Toward High-Risk Minority Youth; Northrop D and Hamrick K. 
Weapons and Minority Youth Violence; Hendrix K and MoUoy PJ. Interventions in Early 
Childhood; Cohen S and Lang C. Application of Principles of Community-Based Programs; and 
O'Donnell L, Cohen S, and Hausman A. The Evaluation of Community-Based Violence 
Prevention Programs. 

Forum on Youth Violence in Minority Communities: Setting the Agenda for Prevention. 
December 10-12, 1990, Atlanta, Georgia. Summary of the Proceedings. Public Health Reports; 
106(3) May/June 1991. 
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less than we would like about how to effectively prevent death and injuries 
resulting from youth violence. Second, what is known about prevention-based 
on innovative efforts by a variety of communities as well as scientific research 
to date-has not been assembled in a clear, concise way for communities to 
use. Third, there has been no accepted locus of responsibility for helping 
communities address the primary prevention of violent death and injuries.^^ 

Thus, the purpose of the gathering was to "(1) summarize what is known about violence 
prevention so that information can be immediately applied by minority communities and (2) 
determine priorities for the evaluation of violence prevention programs so that future 
research can be appropriately targeted. "^^ 

The forum combined plenary sessions and working group meetings. Five working groups 
were organized: (1) principles of community intervention, (2) violence prevention st^-ategies 
focused on minority youth in general, (3) violence prevention strategies targeted toward high- 
risk populations of minority youth, (4) weapons and youth violence, and (5) interventions in 
early childhood. EDC staff provided detailed background papers for each of these sessions, 
as well as a sixth paper that focused On the evaluation of violence prevention programs. 

The units of analysis of the Carnegie survey and conference were programs explicitly focused 
on violence prevention or in which violence prevention was one of several goals (i.e., 
conflict resolution programs seen as a vehicle for violence prevention). Further, these 
programs were considered without regard to whether they relied upon a single intervention or 
a strategy of multiple interventions. 

The unit of analysis for the CDC conference, on the other hand, was for the most part the 
single intervention. Thus, the background paper and working group on violence prevention 
strategies targeted at the general population ol minority youth reviewed educational 
interventions (e.g., interventions to build niale self-esteem, mentoring, teaching conflict 
resolution skills), environmental/technological interventions (e.g., the use of metal detectors 
in schools), recreational interventions (e.g., Boys Clubs), and legal interventions (e.g., youth 
curfews). 

Several of the other working groups shared this focus on interventions. The high-risk 
minority youth group reviewed intei-ventions for gang members, potential gang members, 
drug and alcohol abusers, drug dealers, juvenile offenders, youth with a history of fighting or 
victimization, weapons carriers, youth abused or neglected as children, and a loosely defined 
high-risk group composed of school dropouts and unemployed males. For each group (where 



Public Health Reports, op. cit. 
Ibid. 

Wilson-Brewer R and Jacklin B. op. cit. 
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appropriate) the following types of interventions were discussed: outreach, legal 
interventions, recreational interventions, work/academic interventions, media interventions, 
and environmental interventions/^ 

The weapons and minority youth group explored educational/behavioral, legal, and 
technological interventions/^ The working group on interventions in early childhood 
reviewed a wide range of educational, public awareness, media, and therapeutic 
interventions/^ However, despite this difference in focus from the Carnegie project, the 
background papers reported a similar conclusion: "Based on the lack of evaluation data, it is 
premature to make definitive statements about what types of interventions are most 
effective. "^^ 

The two remaining background papers are discussed below. They focus on the principles of 
community intervention and evaluation of violence prevention programs (the latter was 
distributed to all participants at the CDC forum and was not the subject of a separate 
working group) are discussed below. They also provide a useful context for to discussing the 
conclusions and recommendations from the Carnegie project. 

The following pages contain a summary of the two activities carried out by EDC, through 
Carnegie Corporation's Violence Prevention For Young Adolescents Project: a survey of 
violence prevention programs and a conference on the state of the art in evaluating such 
programs. 

In the final section, we detail the Carnegie project's conclusions and recommendations. At 
the same time, we draw upon some of the important considerations raised at the Forum on 
Youth Violence for the wider context they provide for these conclusions and 
recommendations. 



Northrop D, Jacklin B, Cohen S, and Wilson-Brewer R. op. cit. 
Northrop D and Hamrick K. op. cit. 
Hendrix K and MoUoy PJ. op. cit. 
Wilson-Brewer R and Jacklin B. op. cit., p. 35. 
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THE SURVEY OF VIOLENCE PREVENTION PROGRAMS^ 



METHODOLOGY 

To survey the state of the art of violence prevention programs, a questionnaire and an initial 
list of programs were developed, based upon the staffs prior experience, a literature search, 
and recommendations from experts in the field. The questionnaire solicited basic information 
about each program's goals, target populations, major activities, settings, funding sources, 
staff size, and guiding philosophy. It also asked whether the program and its products had 
been evaluated and what kinds of data have been or are being collected. During a second 
stage of data collection, programs for which evaluations had been conducted were contacted 
for more detailed information. 

Eighty-three violence prevention programs were identified and sent questionnaires in April 
and May 1990. We believe this list includes the majority of existing programs. It is likely, 
however, that despite the best efforts, some programs were missed. Fifty-one of the 
programs contacted completed and returned the questionnaire, for a response rate of 61 
percent. 



FINDINGS 

For the purpose cf this paper, we are concerned solely with what the survey revealed about 
the evaluation of violence prevention programs. For a detailed account of the survey and the 
picture of violence prevention activities around the country that it presents, see the Council 
working paper cited above. 

When respondents were asked the nature of their program evaluation activities, they 
responded as follows: 



^ The survey of violence prevention program.s, summarized here, is reported more fully in 
a separate Council working paper: Wilson-Brewer R, Cohen S, O'Donnell L, and Goodman IF. 
(1991). Violence Prevention for Early Teens: The State of the Art and Guidelines for Future 
Program Evaluation. Washington, D.C.: Carnegie Council on Adolescent Development. 
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Nature of the Evaluations Conducted 



Response 


Number 


Percent 


No evaluation conducted 


8 


16 


Evaluation data unavailable 


7 


14 


Number of subjects counted 


5 


10 


Participant feedback collected 


8 


16 


Monitoring & participant feedback 


12 


23 


Outcome evaluation conducted 


11 


21 



Although staff at the majority of programs surveyed indicated that some kind of evaluation 
activities were ongoing, process evaluation and program monitoring are most prevalent and 
outcome evaluation is relatively rare. Ideally, evaluations should be designed prior to 
program implementation. However, for most part, the evaluation component of programs 
surveyed was either an afterthought or dispensed with entirely because of lack of suitable 
staff and funds. 

For example, 15 programs either conducted no evaluation or collected data that are outdated 
or unavailable. Five tracked only the number of people served. Twenty ask participants to 
complete evaluations at the conclusion of training sessions. Only 1 1 reported conducting 
some form of outcome evaluation. However, in most of these cases the evaluations consist 
of simple pre- and posttest measurements of the attitudes and knowledge of program 
attenders. Often, this method employs unvalidated measures with no control group 
comparisons. In short, there have been only a handful of programs that have been evaluated 
at a level approaching rigorous experimental design. None would meet the most rigorous 
methodologic standards of outcome evaluation. However, definitions of what constitute 
evaluation ranged widely. Nonetheless, there is much to be learned from these attempts at 
evaluation, both about the barriers to evaluating school- and community-based violence 
prevention programs and about how those barriers might be overcome. 

In addition to the survey, case studies were prepared detailing the evaluation experiences of 
11 programs (these are presented in the above-cited related Council working paper). The 
programs listed below were not selected because their evaluations were exemplary in method 
or result. Rather, in each case the evaluation revealed important concerns or barriers that 
could be addressed by the conference participants—and that must be overcome if the field is 
to progress. The programs are 

Boston Conflict Resolution Program (Cambridge, Massachusetts) 

Conflict Management/Peer Mediation Program (Topeka, Kansas) 

Gang Prevention and Intervention Program (Garden Grove, California) 



13 



C J 



ERIC 



The Paramount Plan: Alternatives to Gang Membership (Paramount, California) 

Project Stress Control: School-Based Curriculum (Atlanta) 

Project Stress Control: Youth Development Centers (Atlanta) 

Resolving Conflict Creatively Program (New York City) 

Second Step: A Violence Prevention Curriculum (Seattle) 

Viewpoints Training Program (Santa Barbara, California) 

Violence Prevention Curriculum Project (Newton, Massachusetts) 

Violence Prevention Project (Boston) 

Analysis of the survey results and case studies indicated that quantitative evidence of 
program effectiveness is rare. Such evidence of program effectiveness is limited in a number 
of ways: 

• Although most programs have clearly stated overall goals, frequently these 
goals have not been used to refine specific long- and short-term objectives that 
can be used to inform the evaluation design. Outcomes are often defined 
broadly (e.g., improvements in self-esteem) and are not related to specific 
program objectives and content. This makes it difficult to identify the key 
elements of programs that contribute to desired outcomes. 

• Even when outcomes are clearly defined, they are often specific to a given 
program, as are the indicators used to assess whether tlicy nave been achieved. 
Thus, there is little opportunity to compare results across different programs or 
to build a convincing body of empirical evidence indicating what works best. 

• Interpretation of changes in knowledge, attitudes, and behavior is often limited 
by deficiencies in the evaluation design, including lack of random assignment, 
lack of control groups, the extent to which participants simply repeat the 
answers they have been taught are socially desirable etc. 

• Given the complexity of violent behaviors and violence prevention, there have 
been few attempts to employ multiple measures of impact (e.g., knowledge 
and attitudinal changes, behavioral observations, reductions in disciplinary 
actions). 
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• Because of ease of measurement, the primary findings reported in most 
evaluations are short-term changes in knowledge, attitudes, and self-reported 
behaviors. However, the extent to which these are correlated with or 
predictive of violent behavior, either at the time of the test or over a more 
prolonged period, is uncertain. 

• Given age, developmental, gender, and ethnic differences in the target 
audiences, there has been little attempt to examine the differential effects of 
interventions on subgroups of youth at risk. Little baseline data are collected 
from participants. 

Further, a basic assumption underlying violence prevention programs for young adolescents 
remains untested. Most programs assume-or at least hope-that by intervening witli boys 
and girls during or prior to early adolescence, it is possible to shape attitudes and build skills 
that will reduce their involvement in violence as they mature, and before violent behaviors 
become even more destructive. Yet there are virtu^ly no longitudinal studies of the impact 
of interventions at this age. Program staff express frustration with not being able to follow 
youth to see if their work really does make a difference. However, such longitudinal 
evaluation requires a commitment of resources that goes far beyond what is currently 
available for most programs. 

The limitations of existing evaluations are largely due to the restricted resources that have 
been directed to the evaluation of violence prevention programs, particularly those that were 
spawned at a local level to address an increasing public health threat to the community. 
Whether housed in community centers or schools, these programs have placed an emphasis 
on reaching as many participants as possible. Indeed, raising funds to continue service 
delivery is a constant problem and drain on limited staff resources. 

Resources are not the only limiting factor. Too often, program staff lack the information 
and/or skills to develop and implement either formative or outcome evaluations that would 
inform their work and the field, despite access to large numbers of youth at risk of violence 
and ongoing prevention programs. And many violence prevention programs, with their 
commitment to service delivery and public health perspectives, do not use to their advantage 
the growing body of research on aggressive behaviors that includes a number of carefully 
designed intervention studies. 

Although some programs do make reference to prior research findings and at times even 
include measures of intermediate outcomes drawn from this literature, this is seldom the 
case, based on our review. For example, self-esteem and locus of control scales are often 
used as pre-post measures of changes in attitudes as a result of the program. However, these 
references to the literature are often vague. The question of how to apply research findings 
in the design of programs and their evaluations is typically unaddressed. 
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The need to bridge the gap between researchers and practitioners must be addressed in future 
efforts. At the same time, however, research projects are often limited in time, scope, and 
population reached. Better ways must be found to translate findings and replicate promising 
approaches in community settuigs. 



16 



THE CONFERENCE 



OPENING REMARKS 

David Hamburg, M.D., president, Carnegie Corporation of New York 

This is the third in a series of conferences on this subject that have been held under the 
auspices of the Carnegie Council on Adolescent Development. The first conference, in 
1987, was on a very broad scale. In 1989 we held one focused on what was worthwhile in 
preventative interventions, and on what research base existed for formulating preventative 
interventions. Now we have moved en to evaluation and the use of evaluation in practice. 

Each time I've found it's very easy to say it's timely. On Sunday, July 8th, a lead story in 
the News of the Week in Review section [of the New York Times] was, "A rising tide of 
violence leaves more youths in jail." It begins, "A rising number of the country's 
adolescents are being arrested for violent crimes, with a record 100,000 of them being 
confined in correctional institutions on any given day."^^ 

It's quite horrible if you think about it. It's as if a whole city of 100,000 were in jail today. 
Then the story relates it to poverty, drug use, to single-parent families. It talks about the 
baby boomers' babies and the expected population growth in the next decade and its effect on 
the cost of maintaining and expanding jails. 

Interestingly, it quotes a police official in Boston who says, "We've never had the violence 
among young people that we are experiencing today." He goes on, "Youth violence is rising 
because the problems that cause it aren't being addressed-lack of education, training for jobs 
with a future, housing, recreational facilities, drug treatment," and so on. We are hearing 
more of that sort of thing from the police than we ever did. And for a variety of reasons, 
there is some sense of turning toward preventive interventions by different sectors of society. 
Maybe combining some of the earlier humane compassionate reasons with reasons of cost 
and national interest have led people to ask if this problem might be addressed earlier and in 
a more preventive mode. And that's what we are about. 

We know enough to provide some useful guidelines for preventive interventions to reduce 
substantially the present casualties among both victims and perpetrators of violence. We've 
tried to look at clues from a variety of sources— from direct observation, research, and from 
practice. We've looked at the eticjogy of violence, at risk factors, at development conducive 
to the violent outcomes, including detecting factors in high-risk populations, and a careful 
study of current innovations to provide models of preventive intervention. And now we get 
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to the evaluation of evaluations, looking at differentiated assessments of such efforts and how 
that can be fed back into the system so that we may improve as we go along. 

Now, with respect to adolescent violence, it's clear that it is concentrated in poor and very 
poor-persistently poor-communities, but by no means limited to them. And most 
adolescents are not violent, even in the most poor communities. So, it's necessary to keep 
those differentiations in mind and try to understand how they come about. And yet there is a 
relationship between violence and what William Julius Wilson calls poverty concentration 
areas'^^-there is a very high density of these problems in poverty concentration areas-in 
some ways paradoxically exacerbated by modem technology: the attractive modeling of 
violence on television, the ready availability of drugs, the ready availability of weapons, 
which turn a minor hassle into major violence. All that against a background of certain 
fundamental needs of adolescents that probably haven't changed much over the millennia: a 
need for some sense of worth as a person, some sense of belonging to a group, some skills 
that are compatible with survival, skills that facilitate your getting along in life. 

Society at large, a heterogeneous society, provides a lot of ways to win: many sources of 
self-esteem, many valued skills, many groups for belonging. But how can we make the 
options a reality, make the choices real, provide access to nonviolent choices on an informed 
basis? 

It's a hard problem. It's emotionally charged, complex, and multi-faceted. But we are not 
likely to do very well if we can't get the facts as straight as possible in time. What we have 
tried in this enterprise, and in the work we have supported, is to get the facts straight. Only 
then is it possible to build on the facts-some proven or promising lines of preventive 
intervention at different levels-individual, family, community, society-and even to try to 
think of some that are just beyond our present vision or our grasp. 

Now, when I read the background paper focused on evaluation, there were a few 
implications that struck me. One is the paucity of involvement of universities and major 
research institutes. Mainstream establishments, strong research institutions, seem to have 
quite a marginal involvement. At least there is not much indication that this is high on their 
agendas. And in fact, we know very well they don't have it high on their agendas.. I also do 
not see evidence of a major thrust by the funding agencies, either public or ;.rivate. Perhaps 
that is an unfair assessment, but to me it doesn't look like a big subject : ;. the agendas of the 
major funding agencies. 
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Nor do I see evidence reflected here of an organized constituency, either for research or for 
service, or for evaluation of the service. And yet we all know, and it is borne out in the 
survey data, that this is one of the highest public concerns in this country. 

So, there is some kind of wild, even crazy, discrepancy here between the high level of public 
concern and the lack of an organized constituency or a major thrust on the part of the 
funding agencies, and a very marginal, hands-off involvement of the universities and research 
institutes, as if this were some kind of a low-brow activity. 

How can this national concern be converted into constructive actions? What estimates can be 
made along the most promising lines of inquiry and innovation? Must we wait for definitive 
longitudinal research before we take serious action on a national basis? Can we move now 
on the lessons learned so far from the evaluations, limited though they are, and learn as we 
go to make a long-term commitment to improve these preventive interventions? 

Before I conclude, I want to say something about the generic interventions that are 
addressing the wider context of serious adolescent problems and to underline the predisposing 
factors for a damaging way of life. It is quite clear that, to a considerable degree, these 
adolescent problem behaviors are interrelated. Violence is not usually isolated from other 
aspects of a damaging way of life. And so, violence and related problems have to be, to 
some extent, thought of together in terms of generic interventions to make differences in the 
lives of kids. How do kids grow up, particularly in poverty concentration areas? Can you 
improve the life chances in substantial ways that would diminish the risk of their becoming 
involved in violence and other risks as well? 

That's very important. We need to be asking ourselves, what are the conditions under which 
kids can grow up to be healthy, vigorous, inquiring, constructive people? To what extent 
and in what ways are these conditions lacking in the inner city or other high-risk situations? 
Are there feasible steps that can be taken to diminish damaging influences and improve a 
child's chances? 

Surely a major element in this is the erosion of families and community support. The human 
attachment and constructive models of adaptive behaviors, and mediators for child 
development that are provided under fortunate circumstances by intact, cohesive families, are 
to a certain degree, and often to a great degree, lacking in poverty concentration areas. And 
therefore, we've got to think of other ways either to strengthen them-through parent 
education and social supports for the family— or to substitute experiences that can meet 
essential needs for children and adolescents if it's out of the question for the family to do so. 
That is a fundamental challenge. 

One piece of prevention has to do with making the health care system more adept. It is 
important in its own right, of course, because of the high concentration of all kinds of 
disease and disability problems in the high-poverty concentration areas. But also because a 
health care system, if it functions well, can be a point of entry for various kinds of help. I 

19 



especially emphasize the importance of a new vision of prenatal care, which must reach out 
to bring in poor young mothers early, and which has a strong educational counseling 
component to it. 

The emergency room is obviously important as the first point of contact for some of these 
poor people. It needs to be better linked with educational and social services; to be a 
gateway to needed services for child and adolescent development. And, of course, the third 
particularly crucial point is the school-related health facilities, like the clinics based in our 
schools. 

Another fundamental aspect in addressing this problem is the current move toward school 
restructuring. I think the best example can be found in the intervention of Jim Comer and 
his colleagues in New Haven. That, in my judgment, is the best restructuring effort. It's 
been going on for more than 20 years. I think it is the most important educational reform 
innovation of the past quarter century in this country. It, in essence, is the building of 
supportive bonds between children, parents, and the school staff. The key ingredients 
include a governance team, ways of involving the parents in the school, a mental health 
team. It recombines the more traditional educational approaches with knowledge of child 
development and adolescent development and extends it from the elementary school to the 
junior high school. It began in 1968, and by 19 .;0 there were dramatic changes in academic 
performance and attendance at these inner-city schools that were among the worst when they 
began. And serious behavior problems had diminished enormously.*^ 

Now, this very broad, multi-faceted, humane, practical intervention is being disseminated to 
a number of districts throughout the country, and is being evaluated in a more sophisticated, 
subtle way that will further extend our understanding of what's crucial. And I call it to your 
attention because, in thinking generically about the conditions and influences upon violence 
and other serious problem behaviors in adolescents, reforms in health and education both 
have to be thought about as having real power for this problem and for the remaining 
problems. 

I also want to alert you to a new report, which we had the privilege of supporting. It's 
called Education that Works: An Action Plan for the Education of the Minorities and it came 
out a couple of months ago.** It was put together by a very broad advisory council with 
heavy involvement from a number of minority communities. Basically, it's a consensus on 
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everything we've learned that works, that improves education in disadvantaged minority 
communities. So, I commend it to you. I think it's the best platform we're going to have to 
run on for the next decade in any sort of campaign to improve minority education, and it's 
formulated in a way that's good for all kids. The report is both cognitive and social. It 
deals with social support as a crucial feature of making schools really effective. It touches 
on the problem of drug abuse prevention as well. 

All of these efforts, tapping the hardest problems: disadvantaged minority education, poverty 
concentration areas, substance abuse problems (all of which are related to adolescent 
violence), require that we strengthen our research efforts, particularly evaluation research. 
At the same time, I think we have to face the fact that in matters as complex, difficult, and 
as interrelated as these, the evidence is almost always going to be less than optimal. 

So you can say, and I do hear from time to time, especially in this city, "Well, you know, 
someday, when we know more, we'll be able to intervene efficiently and at a lower cost and 
let's wait for that day." But, we simply have to ask, how much preventable damage is going 
to occur in that length of time? 

So, that's the wrong criteria. You can let the perfect become the enemy of the good. I'm 
all for changing social priorities and changing science policy to address these interrelated 
problems more effectively. But I think the right question to ask is, can we do better than 
we're doing now? If we apply existing knowledge effectively, could we do better than we 
are now doing with respect to the casualties of adolescent violence? And since those 
casualties are so heavy and so serious, I think that there is an urgency that is very special. 

My belief is that we surely do have enough knowledge, evidence, and experience now to 
make things better than we are doing at the present time and to provide conditions in which 
children, especially poor children, can grow up healthy and vigorous, inquiring and problem- 
solving, decent and constructive. 

And of course, we won't be able to prevent all the damage that's inflicted by badly worked 
environments. And if we really address these problems correctly, we will be a lot belter in 
20 years, vastly better in 50 years. But this is the time when we simply have to make the 
synthesis between research, existing best practice experience, to take the richness of recent 
innovations like those under discussion here, and make a serious national effort. And we 
must be determined to utilize the research to improve it year by year, decade by decade-by 
taking advantage of evaluations of other research-to make improvements and build in the 
corrections as we go. 

I'm sure this conference will be a step in that direction. I'm grateful to Ren6e Wiison- 
Brewer, Cheryl Vince, and their colleagues at Education Development Center; to Elena 
Nightingale; and to Del Elliott who have made this meeting possible. And I'm deeply 
grateful to all of you for taking the time out of your busy schedules to come here and help 
all of us help each other in this very difficult and challenging task. 
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Ren^ Wilson-Brewer, senior project director, Education Development Center, Inc. 

This projer:t began to develop over a year ago when the Carnegie Council on Adolescent 
Development held a two-day workshop here in Washington, D.C., on the prevention of 
violence in young adolescents. 

The participants focused on several areas. One was the research on aggression, including 
longitudinal studies on the antecedents to aggressive behavior, and violence in early 
adolescents. They discussed the major approaches to reduce violence in adolescents, with the 
focus on schools. Finally, they considered federal, state, and local policy options with 
respect to the prevention of early adolescent violence, and the roles that foundations, the 
business community, the media, and others can play in addressing the problem. 

An important area that was noted, but not discussed in any detail, was program evaluation. 
Opinion was divided on even the most basic question of how many violence prevention 
programs had been rigorously evaluated. Therefore, one purpose in conducting this project 
was to attempt to answer that evaluation question. A second, and equally important, goai 
was to bring together practitioners, researchers, evaluators, and government representatives 
in an effort to encourage greater collaboration in addressing the problem. 

We hope this working conference will build some new networks. Many of us know one 
another, but I think more often than not, researchers know researchers, practitioners know 
practitioners, and evaluators know evaluators. So, I hope that in this day and a half, we 
really begin to make some efforts to bridge some of those gaps. 

The first step in this project was to identify programs. However, before we could do that, 
we had to reach agreement on what constitutes a violence prevention program. As we noted 
in the working paper, in the broadest sense, any program that seeks to address or ameliorate 
any of the risk factors for violence is really aiding the cause of violence prevention. So, for 
instance, a self-esteem development program will qualify, using this very broad definition. 

However, for the purposes of this review, we developed-with the assistance of our advisory 
committee~a narrower definition. We focused on programs whose explicit goal is or 
includes violence prevention. Using that definition and based on our knowledge of the field, 
we were able to produce a preliminary list of approximately 35 programs. We then shared 
this list with our advisory committee, as well as many of you, and came up with additions to 
the list. 

In quite a few cases, neither we nor others who suggested the names of programs knew 
exactly how they were designed or who the target population was. For some programs, the 
only information we had was name and address. In other cases, it was just a name and a 
city. Sometimes we were successful in locating them and sometimes we weren't. Or we had 
a telephone number, but we called and there was no answer. It just rang, and rang, and 
rang. Or we mailed out correspondence that was returned with no forwarding address. 
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When we sent out the questionnaire, we enclosed our list of programs and asked respondents 
to identify any other programs of which they were aware. As completed questionnaires came 
in, we were then able to add to the program list, with the final total being 83. And of the 83 
programs to which questionnaires were sent, we received a total of 51 responses. 

You may have noticed that we found the most programs in Massachusetts. You might 
conclude that we wanted to make Massachusetts look good. Now, in saying that, I don't 
mean to suggest that having numerous violence prevention programs is a plus; rather that 
there is a great deal of effort being directed toward the problem in our state. It is quite 
possible that there are more programs in Massachusetts. But, I think a more likely 
conclusion is that we have more contacts within our own state and therefore we were able to 
identify several small-scale local programs, the likes of which may exist in other cities as 
well, but we just were not able to get that kind of information. 

So, clearly, we didn't identify all programs, or come up with a definitive answer to the 
question of what is the state of the art of violence prevention in the United States. But our 
goal was to take the first step-to begin to identify the programs, to obtain some basic data, 
and theft" to gain and review program evaluation information. Our hope is that with your 
assistance, we can now move on to the next step, and that is to respond to the information 
gained, to add to it, and then to address several key questions about program development, 
implementation, and evaluation. 

Delbert Elliott, Ph.D., Institute of Behavioral Science, University of Colorado, and 
conference chair 

In the next day and a half, we will focus very sharply upon what it is that we can learn from 
the evaluation of existing violence prevention programs. We will consider the ways in which 
we can improve the kinds of evaluations we do and the quality of programs whose objective 
is the reduction of violence on the part of adolescents. We come from different 
backgrounds, and bring very different perspectives informed by our experiences. 

We will need to listen carefully to one another. This workshop is unique in bringing 
together researchers and practitioners to talk to one another. Very often we don't do that 
well. And the researchers have a different agenda and a different set of objectives. We, the 
researchers, are typically concerned with rules of evidence and scientific kinds of standards 
that often force us to be very hesitant in saying "This is what we know about that has some 
relationship to the termination of violent behavior" or "We know this factor is related to the 
onset of violent behavior. " 

And as a result, there is often a great tension between those who are dealing with the 
immediate problem— with kids standing in front of them who need help~-and researchers who 
say "We don't know enough yet to give you any definitive answer." 
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So, my primary concern is that we share our knowledge and ti7 to focus very specifi'^ally on 
what it is we know about the programs whic!i are currently under way that will help us to do 
fine tuning, to make adjustments, to refocus, in an effort to improve the evaluation of 
violence prevention programs and, through that, the quality of the programs themselves. 

As Renfe mentioned, another important objective is to build new kinds of networks. My 
hope is that we will talk with one another. We will learn what each of us is doing and in the 
process we will establish a whole new set of social networks, so that we can continue our 
dialogue, we can continue to share information, continue to share our findings with one 
another, over the coming years. 

While the focus is upon adolescent violence, I would like to reiterate David Hamburg's point 
that violence does not appear in isolation. It is, in fact, one manifestation of a more general 
adolescent lifestyle, which incorporates a whole series of problem behaviors: minor 
violence, alcohol, substance use, multiple-drug use, precocious sexual behavior, and 
risk-taking behavior in a very general sense. 

So that given an effective program for preventing violence, one is likely to also have an 
impact upon these other behaviors, and that may have some implications for how we evaluate 
programs, because it may very well be that outcomes in which we are interested are much 
broader than just looking at violence, even though violence is one component, and it happens 
to be one which has such serious consequences that we focus upon it more. But from my 
perspective at least, we are looking at a general lifestyle pattern which includes other kinds 
of behaviors. And the implications of that for our treatment strategies and for research I 
think need to be remembered. 

It is also the case that there are in fact multiple etiological paths into violent behavior. There 
is not a single set of circumstances which consistently accounts for this particular outcome. I 
just talked about the problem of adolescent violence with a reporter from People magazine 
who kept asking, "What is the single common denominator?" as if there were some specific 
thing that we could identify that was both sufficient and necessary to produce violent 
behavior. 

Well, I don't believe that's the case. I believe there are multiple etiologies. That has 
implications for the way we approach the onset of violence~or primary prevention-as 
compared to the continuity of violence behavior, the escalation of violent behavior, or the 
termination of violent behavior, each of which may involve different etiological processes, 
different causes, and different circumstances. 

Primary prevention, for that reason, may look different than secondary prevention because 
we may be dealing with different issues and different social, individual factors related to 
those two separate phenomenon, and the awareness that there may be separate phenomenon. 
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I think in the research community we now have a substantial amount of evidence to indicate 
that the etiological factors are different for the continuity of behavior than for the onset or 
the initiation of that behavior, and that has implications for both primary and secondary 
prevention efforts. 

However, from the research perspective, we know very little about the termination of 
behavior. We have been so caught up in trying to account for variation in levels of violence 
that we have just recently begun to focus upon different parameters of violent behavior. Yet, 
termination of behavior is the issue, in many respects, which is most crucial to intervention 
attempts. So, the researchers might comment specifically on what we know about 
termination issues, as they relate to the kinds of programs which we are attempting to-* . 
initiate. 

So, my hope is that we can enter into a careful dialogue, that each of us will listen very 
carefully to one another, so that the dialogue between the research community and those of 
you who are actively involved in working with adolescents to reduce levels of violence might 
be a fruitful one, and that at the end of our time together, we can all go away with a better 
appreciation for the complexities that we are forced to attend to as we address the problem of 
adolescent violence. 



THE PANEL PRESENTATIONS 

Following the opening remarks, five panelists discussed what they have learned from their 
program evaluations. The panelists were 

• Kathy Beland, M.Ed., director of research and development, Committee for 
Children, Seattle 

• Anthony D. Borbon, program director. Turning Point Family Services, Garden 
Grove, California 

• Alice Hausman, M.P.H., Ph.D., assistant professor of pediatrics. Children's 
Hospital, Philadelphia, and evaluator. Violence Prevention Project, Boston 

• Deborah Prothrow-Stith, M.D., assistant dean for community and government 
Programs, Harvard School of Public Health, and co-founder. Violence 
Prevention Project, Boston 

• Ronald Slaby, Ph.D., lecturer at Harvard University, senior scientist at 
Education Development Center, and a designer and evaluator of the 
Viewpoints Training Program. 
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Panelists were asked to consider what they would have done differently in their own 
evaluations, as well as to offer suggestions of more general applicability. Their answers to 
the question "What would you have done differently?" were wide ranging. They touched 
upon the need to hire an outside evaluator and the inevitable tensions that result from vesting 
program and evaluation responsibilities in the same staff. The need to better document the 
program implementation process and its problems was raised. Panelists considered how 
program evaluations that focus on perpetrators of violence can be expanded to look at the 
roles played by bystanders and victims. The question of how to separately evaluate each 
component of a multi-component program was posed by the panelists, as was how to 
properly screen, prepare, and monitor those who are delivering the program. How to 
design, implement, and fund long-term evaluations to effectively measure change was also 
raised. 

A great deal of discussion centered around the issue of developing better methods of 
detecting actual behavior changes-among change agents as well as adolescents. The 
panelists agreed that most programs are no: 'n a position to measure long-term change and 
that this can limit our sense of a program's effectiveness. Program-initiated reductions in 
gang violence, said Anthony Borbon, will not be seen this year or next year. 

Panelists also agreed that the role of those delivering the violence prevention education- 
teachers, counselors, health care providers, community-based program staff, or others-is 
often ignored when behavior change is measured. Providers are very important, said Dr. 
Hausman. They are the first step in violence prevention. If we view the providers as the 
people who are actually going to be effecting change in adolescents, then we really should 
knov; how successful we are in changing those change agents. She explained that it became 
clear through observational data that providers had begun to critically examine how they 
dealt with violence. A focus on the providers and how effective the project was in affecting 
their knowledge and attitudes as well as their behavior toward violence prevention would 
have given us at least some measure of how far along the process the project was in 
achieving violence prevention among adolescents. She suggested that such information could 
serve as a proxy measure of how effective a program could be in the long term. 

Seconding Dr. Hausman' s comments. Dr. Prothrow-Stith emphasized that the transition 
change agents go through, in and of itself, is worth evaluating. The police, the physicians 
and nurses in the emergency rooms, the teachers in the classrooms, the parents, and the 
people in the political system in our communities feel overwhelmed and believe that the 
violence is inevitable. If we can change that attitude-if the provider in the emergency room 
stops treating the violence as inevitable and starts treating it as preventable-if the police start 
treating the violence as preventable, that's an impact that will carry us a long way in actually 
changing behavior. Dr. Prothrow-Stith also stated that in thinking not only about program 
design but also about evaluation methodology, some attention must be placed on finding new 
ways to measure adolescent change witnessed by providers. We must develop methodologies 
to study the impact that providers know occurs in young people, she said. 
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Placing more emphasis on collecting qualitative data (e.g., in-depth interviews with small 
samples of youth) and quantifying that data was discussed. Often a great deal of resources 
and effort are devoted to obtaining limited information on a large number of adolescents. 
What is sacrificed in the process is the opportunity to obtain in-depth information on a 
smaller number of young people. New methodologies for obtaining such information, such 
as the use of drama, was also suggested. A proponent of in-depth interviews, Kathy Beland 
was nonetheless cautious. The interview approach is very time consuming, she said, so the 
nature of our pilot study was, not surprisingly, ruled by time and mon jy. 

Dr. Ronald Slaby discussed the need for more formative research to build bridges between 
research evidence and practice. Describing a program in which his grant included funds to 
hire teachers to provide feedback on a proposed intervention. Dr. Slaby recounted two 
lessons. "We learned that the practitioner is always right about the clarity, relevance, and 
utility of the material. And we also learned that the practitioner is not always right about 
which topic to address, about the ideas behind it, or about how to implement it." Although 
scientific research findings and formulations provide the foundations upon which effective 
interventions can be built, practitioners must be involved if developing interventions, if they 
are to be both implementable and effective. 

Among the other issues raised during this discussion were the need to begin violence 
prevention education early, the possible role of peer leaders in violence prevention programs, 
the impact of dysfunctional families on violent behavior in young people, the failure of 
community institutions to play a major role in teaching socialization skills to youth, the 
prolonged exposure of a large percentage of homicide victims and perpetrators to the 
government welfare/ social services system, and the population of young people who are 
difficult to reach because they move from school to school or have dropped out of school. 

This section of the conference concluded with a special presentation on bullying and 
victimization, given by Dan Olweus, Ph.D., professor in the Department of Psychosocial 
Science, University of Bergen, Norway. A highly-regarded violence prevention researcher. 
Dr. Olweus offered a perspective that stimulated much discussion. 

DR. OLWEUS' PRESENTATION ON BULLY/VICTEM INTERVENTIONS 

Dr. Olweus described a national study of bullying and victimization in Norway and Sweden 
to determine the frequency of bullying. He also described his evaluation of an intervention 
program to prevent bully/victim problems that followed three cohorts of boys (N=9(X)) from 
birth to age 23. The results of his study indicate that bullying is not a kind of childish play 
behavior, but has long-term negative outcomes. 

A student is bullied. Dr. Olweus explained, when he or she is exposed, repeatedly and over 
time, to negative actions of some kind from one or several students. Approximately 15 
percent of students were involved in bully/victim interactions during a given time. He 
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cautioned against relying on popular hypotheses, which may turn out to be false leads. Thus, 
bully/victim behavior was unrelated to class size or to particular external factors such as the 
victim's appearance, or to the bully's failure in school. The effectiveness of his program in 
reducing bullying by 50 percent, he said, was related to several factors: the program is 
theory-driven; it focuses primarily on students' bullying, aggressive behavior rather than on 
assumed factors such as lack of self-esteem; there are consistent and strong sanctions from 
adults regarding unacceptable behavior; and it focuses on several levels of the school system. 

In the question-and-answer period following these presentation, the transferability of Dr. 
Olweus' work to the United States was considered, with a specific focus on how his work on 
bullying and victimization might be related to homicide perpetration and victimization. 
Bullying is part of a more general antisocial behavior pattern, said Dr. Olweus. When you 
follow the former school bullies to age 23, you find a four-fold increase in criminal 
behavior. 



THE DISCUSSION 

The questions addressed by the panelists and other participants provided a focus for small- 
and large-group discussions on July 12 and 13. In addition to the general plenary sessions, 
each participant was assigned to one of three working groups: 

Group I (facilitated by Delber. Elliott): How should violence prevention programs be 
evaluated? 

Group II (facilitated by Mark Rosenberg, M.D.): How can practitioners, researchers, 
and evaluators connect and then collaborate as truly equal partners in program 
development and evaluation? What are the major barriers to collaboration? 

Group III (facilitated by Paul Bracy): What role should substance abuse and other 
problems play in violence prevention programs? Should violence prevention be taught 
in a specific domain or domains, or in its own right? 

Each group was asked to develop specific recommehdations for discussion among all 
participants. This general consideration .ind further development of the recommendations 
was followed by a final plenary session devoted to the suggestions for implementing the 
recommendations. 

Both because the topics assigned to the working groups were not, by their nature, mutually 
exclusive, and because of the broad range of the conferees' expertise and experiences, many 
of the same themes were touched on and later integrated and expanded upon when the groups 
came together. Indeed, many of these same themes were foreshadowed in Dr. Hamburg's 
opening remarks. Therefore, rather than present a purely chronological summary of the 
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discussion, we will consider the v/ork of the conference through the major themes that 
emerged and informed the recommendations. 

The Current State of the Evaluation of Violence Prevention Projects 

None of the conference participants was sanguine about the current state of the evaluation of 
violence prevention projects. There was general agreement among researchers, practitioners, 
and representatives of foundations and federal agencies that too few evaluations have been 
completed and that even the small number of more rigorous attempts leave many critical 
questions unanswered. Voicing a common concern. Dr. Olweus said. It's really disturbing to 
read that thousands and thousands of students have taken certain programs and there is no 
evidence at all that they really work. Participants also referred to our inability to measure 
whether any program has reduced violence or, failing that, to develop acceptable 
intermediary measures of outcome. At the same time, concerns were raised that evaluation 
is often perceived as a political exercise to justify a program's continuance or termination. 
"Evaluation is often perceived as a straight up or down vote," said Dr. Hamburg, "and if 
you put it that way, any program person in his or her right mind would reject it." 

Nor was the need for improved evaluation couched only in terms of new program 
development. As Dr. Nightingale and other participants noted, the need to improve and 
expand the evaluations of promising cunent programs is paramount. Finally, the consensus 
on the need for evaluation was voiced by Dr. Prothrow-Stith: "How do you, in the 
formative phases of an area such as violence prevention, foster programs that will be worthy 
of evaluation and develop methodologies that can eventually demonstrate outcomes, without 
killing the buds that might flower in the future?" 

What Programs Should Be Evaluated? 

There was broad agreement that it is neither practical nor scientifically necessary to fully 
evaluate every intervention program. Certainly, high costs and the limited resources and 
expertise available militate against doing so. It was suggested that attention be focused on 
model programs, the results of whose rigorous evaluations could be generalized to new 
locations and target populations. 

However, in restricting full outcome evaluations to a smaller number of carefully selected 
model programs, it was noted that all programs require ongoing process evaluations to assist 
in monitoring their implementation. "You have to have an evaluation that will at least allow 
you to determine whether there is program integrity," said Dr. Elliott. And many programs. 
Dr. Slaby noted, could benefit from conducting formative evaluations with members of their 
target audiences. 
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Impediments to Evaluation 

"If you look at the objectives of many programs," said Dr. Lydia O'Donnell, "you will find 
that they are not very different from other programs in other cities or regions of the country. 
But what they measure is so different that you can't make any comparisons among them. 
We need to help programs to develop a more consistent set of standards for evaluation." 
Consistency is required at the very beginning, when programs define their goals and 
objectives. One of the problems we have is that, if a program doesn't define what it means 
by violence, we don't know what we are measuring, said Dr. Leonard Eron. Participants 
noted that existing programs target a broad range of violent behaviors and, for clarity, 
adopted the following statement: 

Because the field of violence prevention is extremely broad in scope, we have 
adopted the following statement: The primary focus of the violence prevention 
programs under consideration is the reduction of physical violence intended to 
cause harm. We do acknowledge the existence of other forms of violence that 
are appropriate subjects for intervention and evaluation, but they are not the 
primary focus of this conference. 

Participants recommended that each program must carefully and explicitly define what 
behaviors it has targeted, and then tie its evaluation to goals and outcomes. They noted that 
the lack of resources to support evaluation is a consistent problem, but that even if funds 
were widely available, there are other, methodologic difficulties. 



The Need for New Approaches to Evaluation 

Several compelling issues were raised about the nature of evaluation research and its 
application to violence prevention programs. The difficulty of searching for short-term 
results in a long-term process (behavior change and violence reduction) was noted. When 
you evaluate gang prevention efforts, Anthony Borbon reminded the group. It will take 
anywhere from five to ten years for us to track individuals in the program, to see if they 
graduate, if they join mainstream society. 

At the same time-controlled comparisons-the most common and rigorous of evaluation 
techniques may not be applicable to violence prevention evaluations, both for practical and 
ethical reasons. "Researchers will always push you toward experimental designs because 
those are the most powerful," cautioned Dr. Elliott, "but those designs imply a no-treatment 
alternative and it's very difficult to justify that in ethical terms." And Dr. Prothrow-Stith 
recounted the difficulties experienced by the Violence Prevention Project, which focused on 
two Boston neighborhoods, when other communities that might have served as controls began 
requesting services as well. 

However, Drs. Elliott, Hausman, and Donald Schwarz, M.D., as well as other researchers, 
explained that a variety of new or long-accepted evaluation techniques can be drawn on to 
overcome the barriers imposed by traditional experimental designs. For example, Dr. 



30 



Olweus described the age cohort design utilized in his bully/victim intervention. In this 
design each cohort serves as both treatment and control groups in different comparisons. 

In addition to the research design, there was concern about the nature of the evidence 
gathered for evaluation purposes. Describing her own effort to evaluate the Violence 
Prevention Project, Dr. Hausman said, "We placed a lot of emphasis on getting a large 
amount of data on a large number of adolescents. What we sacrificed was getting some 
really good, qualitative information on a smaller number of kids. I would like to have done 
in-depth interviews to really understand how a smaller sample really incorporated violence 
prevention education in their lives." Indeed, the survey of violence prevention programs 
revealed that practitioners often collect anecdotal and other qualitative evidence from 
adolescents or their families but see no way to integrate the material with their quantitative 
evaluation data. 

Based on this discussion, participants concluded: 

We stress the need to use qualitative data, culturally sensitive measures, and 
nontraditional evaluation designs to reduce the difficulties and ethical dilemmas 
that can be imposed by strict experimental designs. We urge the use of 
multi-level evaluation strategies that include learning from children, parents, 
teachers, and others. Evaluations must incorporate multiple measures of the 
reduction or increase of violence, including data collected from many sources, 
such as self-reports, observations of others, and criminal justice data. 

What Are the Outcomes and How Do We Measure Them? 

Perhaps the thomiest-and arguably most important-methodologic questions revolved around 
what an evaluation of a violence prevention program should measure. "If we are looking at 
the reduction of violent behavior as the outcome of these programs," asked Dr. Elliott, "how 
can we specify the intermediate, intervening, targeted objectives that are theoretically 
connected to the outcome?" Measuring knowledge and attitudes, while often a useful part of 
an evaluation, cannot demonstrate a reduction in violence. 

Betty Hamburg, M.D., offered an analogy from medicine. Preventing heart attacks and 
stroke, she noted, depends on such intermediate objectives as weight reduction, smoking 
reduction, increased exercise, dietary changes, etc. Each of these is an objective that can be 
measured in its own right (although the causal relationship of each of these factors was not 
necessarily known even though the association was). 

Without specifying intermediate objectives as steps toward violence prevention, our ability to 
detect any change is severely reduced. Particularly over short time periods, expected effects 
would not be great, available statistical measures may not be sensitive enough, and target 
populations may be too small. 
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A variety of possible intermediate objectives were proposed, although not discussed in detail. 
They included changes in school climate or at the classroom level, changing rates of teenage 
pregnancy, and reductions in the amount of graffiti in areas of gang violence, among others. 

One often overlooked intermediate objective that was discussed in greater detail was changes, 
not in the target population itself, but in the change agents (e.g., teachers) through whom 
that population is reached by a project. "Children grow up in situations they didn't create 
but are reacting to," noted Paul Bracy, "We try to intervene with them and to measure the 
changes as if they are the problem. But adults are responsible." 

In reviewing the evaluation of the Boston Violence Project, Drs. Hausman and Prothrow- 
Stith emphasized the importance of evaluating changes in the agents of change. "You can't 
expect much change in the adolescents if the teachers themselves are not convinced," 
cautioned Dr. Hausman, adding, "Violence is a very personal issue and often the providers 
have to go through major changes in how they think about and deal with violence 
themselves." 

Nor was the goal merely to improve evaluation methodology. "We must understand," said 
Dr. Prothrow-Stith, "that we are so early in this game of violence prevention that if we can 
actually change the attitudes and practices of service providers, that we've gained a 
considerable amount and are an important step closer to changing behavior. " 

Fostering Collaboration between Researchers and Practitioners 

The most frequently heard suggestion to improve the relationship between program 
practitioners and evaluators was to bring them together early— from the very development of 
the intervention, if possible. If evaluators and program people are going to be friends, they 
have to work together from the beginning. You have to set up a program so that it can be 
evaluated, noted Dr. Rosenberg. As an example, Dr. Slaby described the Children's 
Television Workshop Model (from which such successful programs as "Sesame Street" were 
developed). "You bring all of the components together from the very beginning and get 
them to work out a common language," he said. The lack of such a common language, not 
only among researchers and practitioners but among all of the disciplines involved in 
violence prevention, was frequently noted by participants. 

Such collaboration, it was urged, should extend from the inception of a project to the 
conclusion of its evaluation. Not only should evaluators be involved in designing an 
intervention, participants stated, but practitioners should co-author reports and journal 
articles. Carl Bell, M.D., urged that evaluators work closely with program staff to insure 
that interpretation of the data is informed by the program's experience. And throughout the 
implementation of a project, an evaluator's insights can be useful in interpreting and using 
process data to monitor and adjust activities. Noting that such collaboration can be 
expensive. Dr. Betty Hamburg said, "I don't think that funders have appreciated [its] 
importance." 
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Providing Assistance to Practitioners 

A variety of specific products and activities were suggested to assist practitioners in 
improving evaluation skills and in obtaining expert assistance. These are summarized in the 
conclusions and recommendations of the conference. 



FINDINGS FROM THE DISCUSSION 

As is to be expected of any such gathering, no matter how circumscribed the focus, the 
discussion was wide ranging and thought provoking. For the purposes of this summary, 
however, attention is placed on those findings that most directly address the evaluation of 
violence prevention programs for young adolescents. 

The general sessions and three conference workshops resulted in the following findings: 

• Each program must carefully and explicitly define what behaviors it has 
targeted, and then tie its evaluation to goals and outcomes. Evaluations and 
programs must be theoretically informed. A rationale for the selection of 
immediate targeted objectives and their relationship to the expected outcomes 
must be specified. If we are looking at a reduction in violent behavior as an 
outcome, it is necessary to specify intermediate, intervening, targeted 
objectives that are theoretically connected to the outcome. In addition, both 
positive and negative unanticipated outcomes, or the side effects of an 
intervention, must also be taken into account. 

• Different levels of evaluation are necessary for different programs. Not all 
programs require a full outcome evaluation, but all programs require a process 
evaluation to assist in improving implementation. We must identify or develop 
model programs for which model evaluations can be carried out. 

• We stress the need to use qualitative data, culturally sensitive measures, and 
nontraditional evaluation designs to reduce the difficulties and ethical dilemmas 
that can be imposed by strict experimental designs. We urge the use of 
multi-level evaluation strategies that include learning from children, parents, 
teachers, and others. Evaluations must incorporate multiple measures of the 
reduction or increase of violence, including data collected from many sources: 
self-reports, observations of others, criminal justice data, etc. 

• Program evaluation must be a collaboration from the earliest stages, making 
explicit the conceptualization of the program, the theoretical rationale that will 
drive the program components, and an appropriate interpretation of the 
findings. 
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We must assist programs to develop evaluation skills and to use expert 
assistance to maximize the use of limited resources. We need to pool our 
information on appropriate measures and evaluation methodologies. We must 
develop mechanisms to encourage and sustain ongoing relationships between 
practitioners and researchers. 

To assist practitioners and advance the state of the art, it is necessary to 
develop a handbook for violence prevention program evaluation. Modelled on 
the National Research Council document Evaluating AIDS Prevention 
Programs, this handbook should be developed in collaboration with a panel of 
evaluators and practitioners. 

We must address the personnel needs in this area by increasing the supply of 
researchers and evaluators in the area of violence prevention, as well as the 
availability of stable, long-term funding for programs. It is especially 
important that we increase the number of minority researchers, practitioners, 
evaluators, and social scientists. 

Violence prevention is a multi-disciplinary field in which collaboration among 
the disciplines has not yet been developed. We must support exchanges of 
knowledge among fields and the development of a common language and a 
common body of knowledge. 

The consistent association between substance use/abuse and violence demands 
that substance abuse prevention education be integrated into violence 
prevention programs. We must also ensure that violence prevention education 
is included in programs that focus on substance abuse prevention. 

We must promote programs that provide young people with opportunities for 
skills development that can lead to economic advancement. Although not 
violence prevention programs per se (e.g., job training, community service), 
they ofrer participants prosocial and economically profitable alternatives to 
violence. 

To address the wider context of violence, it is important to develop and fund 
interventions that consider the multiple causes and target not only the 
adolescent but also the family, community, and public policy. 

Violence prevention as a field and in its specific activities remains poorly 
understood. We must legitimize and "sell" the field of violence prevention by 
focusing on violence as one of the most serious, life-threatening, 
injury-producing dysfunctional forms of problem behavior. However, its 
connection with other variables (e.g., substance abuse, weapon availability) 
should not be ignored. 



CONCLUSIONS AND RECOMMENDATIONS 



Between 1987 and 1990, staff at EDC participated in two reviews of the state of the art in 
violence prevention interventions: Injury Prevention: Meeting the Challenge and the 
background papers prepared for the CDC's Youth Violence in Minority Communities: A 
Forum on Setting the Agenda for Prevention. The Carnegie conference on violence 
prevention for young adolescents, however, was perhaps the first attempt to systematically 
address the question of how these programs have been evaluated. As such, the project, and 
the two resulting Council working papers,"*^ deal primarily with the process and content of 
the evaluations themselves. Much of the conference discussion focused on what could be 
learned from these evaluations to inform and improve future efforts. 

During the conference another and different question was posed by Dr. Elena Nightingale, 
special advisor to the president of Carnegie Corporation. Of all of the programs 
investigated, she asked, were there any that in whole or in part warrant greater 
dissemination? Which of these programs, she was in effect asking, had been subjected to 
rigorous process and outcome evaluation and been found to be effective in preventing 
violence among young adolescents? 

The direct answer is that none of the programs surveyed has been so evaluated. Therefore, 
given the limitations of existing evaluations of violence prevention programs, it would be 
premature to come to closure about what works and what doesn't. That does not mean that 
we lack any indicators of which programs are promising. Nor does it mean that we should 
abandon programs that are currently reaching large numbers of persons. It does raise the 
question of what steps can be taken to improve our understanding of these programs and 
their effectiveness. And it poses two central challenges for violence prevention practitioners, 
researchers, and funding agencies: how to identify current, promising programs and 
strengthen their efforts while simultaneously designing new programs so well conceived and 
carefully implemented as to be worth the expenditure of resources for the kind of full-scale, 
rigorous evaluation that has been so rare. 

To search for answers to these questions within the activities described in this paper alone 
would be insufficient. Evaluation is an important conponent— indeed, a critical one-for any 
project focused on achieving social change. But the questions we must answer are about 
design and implementation of approaches as well as evaluation. 



This paper is complemented by a separate Council Working Paper focused on the survey 
of violence prevention programs carried out in preparation for the Carnegie Conference on 
Violence Prevention for Early Adolescents. See Wilson-Brewer R, Cohen S, O'Donnell L, and 
Goodman IF. (1991). Violence Prevemion for Early Teens: Vie State of the Art and Guidelines 
for Future Program Evaluation, Washington, D.C.: Carnegie Council on Adolescent 
Development. 
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To truly understand both whether a violence prevention program was effective and the 
likelihood that it could be replicated, we must know the following: 



• How was the program developed? What theoretical or empirical body of 
knowledge informed its creation? 

• How were the intervention(s) and target populations selected? What 
indications existed that the fit between intervention and population v/ere the 
most appropriate? What, for example, was the role of formative evaluation? 

• What were the program's objectives and how were they carried out? What 
was the role of process evaluation? 

• What effects did the program produce? What was the role of outcome 
evaluation? 

Taken together, the Carnegie conference and the Centers for Disease Control's forum 
provide the broader view that is necessar>'. In addressing Dr. Nightingale's questions and in 
providing the most useful context for the Carnegie conference's specific recommendations 
reported below, we will draw upon some of the points raised in the CDC forum materials 
cited above. 

In addressing the application of principles of community-based programs to violence 
prevention in minority communities, the EDC background paper drew upon several sources. 
These included the general injury prevention program model described in Injury Prevention: 
Meeting the Challenge, a model for diffusing health information in minority communities,^^ 
and CDC's Planned Approach to Community Health Program (PATCH). 

The process for developing community violence prevention programs described as 
background for the CDC forum includes a series of elements: developing leadership, 
identifying the problem, devising a strategic plan to address the problem, translating the plan 



National Heart, Lung, and Blood Institute. (1987). Strategies for Dijfusing Health 
Information to Minority Populations: A Profile of a Community-Based Diffusion M?uel. 
Washington, D.C.: U.S. Department of Health and Human Services. 

Centers for Disease Control, (n.d.). Planned Approach to Community Health. Atlanta: 
U.S. Public Health Service. See also: Nelson CF, Kreuter MW, Watkins NB, and Stoddard 
RR. (1986). A Partnership Between the Community, State, and Federal Government: Rhetoric 
or Reality? International Journal of Health Education^ 5:27-31. 
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into action by designing an intervention or program of interventions, implementing that 
intervention successfully, and evaluating its effects.^^ 

It should be stressed-and the point was emphasized by participants in the Carnegie 
conference-that although evaluation is always named last in the series of steps (e.g., design, 
implementation, and evaluation), it must be seen as zn integral part of the process. 'The 
time to plan an evaluation," concluded the National Committee for Injury Prevention 3rd 
Control, "is in the very earliest stages of program design. "^^ "It's really unconscionable," 
commented conference participant I^eonard D. Eron, Ph.D., "to start any kind of program 
without some idea of how you're going to evaluate its results." 

We must, however, return to the initial question: What can the evaluations of existmg 
violence prevention programs tell us about what directions are promising? If the evaluations 
alone are insufficient to identify promising programs, then what else do we need to know to 
improve our understanding and move the field forward? 

In its 1987 survey of the state of the art in injury prevention, the National Committee for 
Injury Prevention and Control studied a wide range of existing countermeasures against all 
forms of injury, most of which had not been rigorously evaluated. In categorizing these 
interventions, the committee used the following terms: "proven effective," "promising," 
"ineffective," or "unknown or insufficiently studied." For each intervention, a 
recommendation included both a statement of efficacy and suggestions for further use. Thus, 

• an intervention that is proven effective (e.g., bicycle helmets) should be used 
and the program should be routinely monitored (i.e., a process evaluation is 
necessary, but not an outcome evaluation) 

• an intervention that is promising (e.g., raising excise taxes on alcohol to 
reduce its availability) should be used, monitored (i.e., through process 
evaluation), and its outcomes should be rigorously evaluated 

• an intervention that is ineffective or counterproductive (e.g., painted lines at 
crosswalks) should not be used 

• an intervention whose efficacy is unknown or insufficiently studied (e.g., 
designated driver and safe-ride programs) should be the subject of further 
research.^ 



Cohen S and Lang C. op. cit., p. 6. 

National Committee for Injury Prevention and Control, op. cit., p. 78. 
National Committee for Injury Prevention and Control, op. cit., p. 113. 
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The categories used by the committee were deliberately broad because the evidence available 
on which to make judgments about interventions was often extremely limited. In many 
cases, only a single article (or unpublished document) existed about a specific intervention. 
Committee working groups worked hard to find a reasonable basis on which to assign each 
intervention to a category. The number of proven effective or completely ineffective 
interventions was small. Much remains to be learned about the others. 

An attempt to describe the 51 projects surveyed in these terms reveals that the vast majority 
are of unknown or insufficiently studied value in preventing violence among young 
adolescents. As most of these projects have not been evaluated, this conclusion is not 
surprising. 

However, based upon existing-albeit incomplete-evaluation data, it is possible to identify a 
smaller number of projects whose assessments suggest that (1) their results to date have been 
promising and (2) further study could yield valuable information about the development, 
implementation, and evaluation of future programs. These projects include 

• The Paramount Plan: Alternatives to Gang Violence (Paramount, California) 

• Project Stress Control School-Based Curriculum (Atlanta) 

• Resolving Conflict Creatively Program (New York City) 

• Second Step: A Violence Prevention Curriculum (Seattle) 

• Viewpoints Training Program (Santa Barbara, California) 

• Violence Prevention Curriculum Project (Newton, Massachusetts) 

• Violence Prevention Project (Boston)^^ 

These projects should be continued and should be monitored through process evaluation and, 
most important, new or expanded outcome evaluations. At the same time, a study of their 
conceptual, theoretical, and research bases should be undertaken to answer a series of 
questions: What are the behaviors that the programs are intended to change? What are the 
probable or known causes of those behaviors and to what extent did that research base 
influence the selection of interventions and target populations? Which specific considerations 
informed the selection of interventions? To what extent does the community in which the 
program operates feel any sense of ownership of violence prevention, in general, and the 
program specifically? 



Descriptions of these projects are contained in Wilson-Brewer R, Cohen S, O'Donnell L, 
and Goodman IF. op. cit. 
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Only by answering such questions will it be possible to identify model projects that could be 
disseminated widely without the need to conduct a full outcome evaluation in each new 
setting. And, only by asking such questions will it be possible to close the gap-identified by 
several— between violence prevention research in the behavioral sciences and practice in the 
field. As commented Ron Slaby, "We wouldn't think of dealing with an AIDS prevention 
program, at this point, without being informed of the research base about how AIDS in 
contracted. With AIDS, our practice is guided by the state of research evidence. That is 
often not the case with regard to violence prevention." 

With these considerations in mind, we present the following recommendations for action. 
They are of two types. The program recommendations were developed by the authors during 
a reanalysis of the survey data. The products and activities, on the o^her hand, were 
suggested by the working groups and refined during subsequent plenary discussion. 

PROGRAM RECOMMEND/^ TIONS 

1. Funding and technical assistance should be made available to review the programs 
listed above with a view to determining which among them are sufficiently promising 
to make the expenditure of resources for new or additional outcome evaluations 
worthwhile. 

2. The provision of resources for the expanded evaluation of promising programs is a 
task beyond the capacity of any single funding source. Because this need is both 
national and resource-intensive, the effort should be coordinated by a consortium of 
government agencies and foundations. 

3. The identified promising projects should be used as laboratories for the development 
of intermediate outcome measures, for the validation of existing measures, and for the 
development of new evaluation methods. Such efforts will require the collaboration 
of program practitioners, individual researchers, and public and private research 
institutions. 



SPECIFIC PRODUCT RECOMMENDATIONS 

1. A handbook on violence prevention evaluation should be developed as a collaborative 
effort of evaluators and practitioners. Such a volume is necessary to aid violence 
prevention practitioners and to advance the state of the art. Such a handbook would 
assist program staff to carry out evaluation tasks, as well as to collaborate with 
evaluation research specialists in the development and implementation of evaluation 
designs. It should contain information on available resources, including a list of 
evaluation researchers and their specific areas of expertise. General purpose primers 
on evaluation fail to provide good outcome and process measures that are relevant to 
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violence prevention programs or identify qualitative evaluation strategies. A useful 
model for such a handbook is the recent document, Evaluating AIDS Prevention 
Programs, published by the National Research Council." 

2. A primer and catalog on the identification and use of culturally sensitiv'e measures in 
conducting formative and outcome evaluations should also be created. This document 
will review the evidence on how cultural differences in values and behavioral norms 
among ethnic and racial groups can affect the design and implementation of violence 
prevention interventions. It will identify measures that have been used successfully 
w-th different groups, as well as detail the critical steps in deve^-^ping new measures. 

ACTIVITIES 

1. Convene an annual meeting of violence prevention practitioners to improve 
communication and problem solving. 

2. Conduct a summer institute in program development and evaluation for violence 
prevention practitioners, to help them develop and enhance skills in carrying out 
evaluation tasks, and work effectively with speci lists to plan evaluation designs. 

3. Develop interdisciplinary research centers to focus on the evaluation of violence 
prevention programs. 

4. Increase the outreach to and recruitment of minority students and faculty and provide 
scholarships to encourage study in the field of violence, especially in research and 
evaluation. 

5. Conduct rigorous evaluations of model programs already underway. 

Each of these recommendations speaks to a specific need that must be met if we are to 
improve the evaluation of violence prevention programs and, through improved evaluations, 
affect the development and implementation of more effective strategies. No single 
recommendation is sufficient, however. 

If, as Dr. Hamburg suggested, there is a need to bring coherence to the nascent national 
effort to prevent violence among young adolescents, that is no less true of program 
evaluation. Increasing the capacity of program practitioners to learn from each other, to 
understand and benefit from evaluation research, and to collaborate fruitfully with 
researchers are related priorities. So too is the need for researchers to learn from violence 



Coyle SL, Boruch RF, and Turner CF. (eds). (1991). Evaluating AIDS Prevention 
Programs (Expanded Edition). Washington, D.C.: National Academy Press. 
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prevention practice, not only to advance the development of theory, but to help develop new 
and innovative strategies for program evaluation. The identification of intermediary 
objectives-short-term gains that presage the longer-term goal of preventing violence-is a 
particularly critical need. How to organize these pressing tasks remains to be considered. 
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